System, method, and computer program for transmitting face models based on face data points

ABSTRACT

A system, method, and computer program are provided for receiving face models based on face nodal points. In use, a real-time face model is received, wherein the real-time face model includes one or more face nodal points. Real-time face nodal points are received, including additional one or more face nodal points. The real-time face model is manipulated based on the real-time face nodal points.

CROSS-REFERENCES TO RELATED APPLICATIONS

The present application is a continuation of, and claims priority toU.S. patent application Ser. No. 17/543,519, titled “SYSTEM, METHOD, ANDCOMPUTER PROGRAM FOR TRANSMITTING FACE MODELS BASED ON FACE DATAPOINTS,” filed Dec. 6, 2021, which in turn is a continuation of, andclaims priority to U.S. patent application Ser. No. 17/108,867, titled“SYSTEM, METHOD, AND COMPUTER PROGRAM FOR TRANSMITTING FACE MODELS BASEDON FACE DATA POINTS,” filed Dec. 1, 2020, which in turn is acontinuation of, and claims priority to U.S. patent application Ser. No.16/547,358, titled “SYSTEM, METHOD, AND COMPUTER PROGRAM FORTRANSMITTING FACE MODELS BASED ON FACE DATA POINTS,” filed Aug. 21,2019, which in turn claims priority to and the benefit of U.S. patentapplication Ser. No. 16/206,241, titled “SYSTEM, METHOD, AND COMPUTERPROGRAM FOR TRANSMITTING FACE MODELS BASED ON FACE DATA POINTS,” filedDec. 30, 2018, which in turn claims priority to and the benefit of U.S.Provisional Patent Application No. 62/618,520, titled “SYSTEM, METHOD,AND COMPUTER PROGRAM FOR TRANSMITTING FACE MODELS BASED ON FACE DATAPOINTS,” filed Jan. 17, 2018, all of which are hereby incorporated byreference for all purposes.

FIELD OF THE INVENTION

The present invention relates to transmitting a face model, and moreparticularly to transmitting a face model based on face data points.

BACKGROUND

Current real-time communication systems (e.g. video conference, etc.)require high bandwidth and consume a large volume of data duringoperation. To enable fluid video conferencing capabilities, such systemsgenerally require a user to choose between lower data rates (lowerquality) or higher data rates (higher quality). In another instance, arelatively unreliable network connection for a connected user maydirectly cause a lower rate video stream (and hence a lower qualitystream) to be sent.

There is thus a need for addressing these and/or other issues associatedwith the prior art.

SUMMARY

A system, method, and computer program are provided for receiving facemodels based on face nodal points. In use, a real-time face model isreceived, wherein the real-time face model includes one or more facenodal points. Real-time face data points are received, includingadditional one or more face nodal points. The real-time face model ismanipulated based on the real-time face nodal points.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates an exemplary method for transmitting face modelsbased on face data points, in accordance with one possible embodiment.

FIG. 2A illustrates a method for determining face data points, inaccordance with one embodiment.

FIG. 2B illustrates one or more parameters, in accordance with oneembodiment.

FIG. 2C illustrates a method to correlate phonetics/emotions with animage depth map, in accordance with one embodiment.

FIG. 2D illustrates a method to manipulate a face model, in accordancewith one embodiment.

FIG. 3A illustrates a digital photographic system, in accordance with anembodiment.

FIG. 3B illustrates a processor complex within the digital photographicsystem, according to one embodiment.

FIG. 3C illustrates a digital camera, in accordance with an embodiment.

FIG. 3D illustrates a wireless mobile device, in accordance with anotherembodiment.

FIGS. 3E illustrates a camera module configured to sample an image,according to one embodiment.

FIG. 3F illustrates a camera module configured to sample an image,according to another embodiment.

FIG. 3G illustrates a camera module in communication with an applicationprocessor, in accordance with an embodiment.

FIG. 4 illustrates a network service system, in accordance with anotherembodiment.

FIG. 5 illustrates a network architecture, in accordance with onepossible embodiment.

FIG. 6 illustrates an exemplary system, in accordance with oneembodiment.

DETAILED DESCRIPTION

FIG. 1 illustrates an exemplary method 100 for transmitting face modelsbased on face data points, in accordance with one possible embodiment.As shown, a first image is received (see operation 102), and at leastone face associated with the first image is identified (see operation104). In one embodiment, identifying the at least one face may includesegmenting the first image into regions, and determining that one ormore of the regions comprises a face. In another embodiment, identifyingthe at least one face may include a classifier engine, such as a neuralnetwork classifier engine, identifying a face region in the first image.

In various embodiments, the first image may be received via a camera,via an application, from a local storage (e.g. device memory), from aremote storage (e.g. cloud, external server, etc.), etc. As such, thefirst image may be the result of a recent capturing by a camera device,or may be the result of retrieving or transferring of a previouslycaptured and stored image.

Next, a face model is created of the at least one face by determining ageometric structure of the at least one face, wherein the face modelincludes one or more face data points (see operation 106), and the facemodel is transmitted (see operation 108). In the context of the presentdescription, a face model includes face data points from which a facemay be visually constructed. In one embodiment, a face model may includea mapping of facial features (e.g. face nodal points) to the geometricstructure. Additionally, face data points may include facial features(e.g. distance between the eyes, width of the nose, depth of the eyesocket, shape of the bone structure, width of jaw, etc.), which may bestored as face nodal points.

In the context of the present description, a face model may include ageometric mesh or collection of geometric objects. The geometric objectsmay include two-dimensional vertices or three-dimensional vertices. Theface model therefore provides sufficient visual detail for rendering ofa face (e.g., a human face or an avatar face). Additionally, face datapoints may include any number of data points based on the face model.

Further, a real-time stream of the at least one face is provided (seeoperation 110), and a real-time face model of the real-time stream usingthe face model is determined (see operation 112). Next, the real-timeface model is transmitted. See operation 114.

In the context of the present description, a real-time stream mayinclude a video chat and/or conference call. Additionally, the real-timestream may function in near-real time such that data transferred isprocessed within a set time (e.g. within 500 milliseconds).Additionally, the real-time stream may function as a streaming of datasuch that data is continuously transferred with minimal latency. In aseparate embodiment, the real-time stream may include a delay (e.g. <=5seconds) due to buffering, a change in network (e.g. switch from WiFi tocellular network), etc.

Further, a real-time face model may include real-time face data pointsfrom which a face may be rendered. In this manner, as an individual'sface changes in the course of a conversation, such changes may beobserved and analyzed for movement by a first device and transmitted toa second device; the second device configured to render and animate thereal-time face model.

In one embodiment, face data points may be used to generate a stream ofvideo frames that depict a face speaking by modifying a face modelaccording to speech or expression. For example, a face model may be sentfrom the first device to the second device. A video stream may beestablished (e.g., initially) from the first device to the seconddevice. In one embodiment, rather than sending all data associated withvideo frames of the face of the user in real time, the data associatedwith the face data points may be sent instead (thereby reducing databandwidth), and such face data points sent from the first device may beused to manipulate the face model at the second device. As such, if theuser at the first device raises an eyebrow, rather than sending videoimage data for the entire frame or even motion estimated image data forthe eyebrow, the face data points for the eyebrow may be sent instead(e.g. move point 1 up, move point 2 to the side, etc.) and the facemodel at the second device may be modified and rendered in real time totrack and depict the raised eyebrow.

As another example, a first user may capture an image of the first user.Such image may be processed to determine a face model including facedata points. In one embodiment, the face model may include both thefirst image and the face data points. Such face model may then betransmitted to a second device, which may then open the face model todisplay the image of the first user. After establishing a video streamwith the first user, the second device of the second user may thenmanipulate the original face model such that changes in face nodalpoints of the first user may be transmitted. In contrast to sending theentire image data (e.g. video frames) of the first user, the first usermay then need to only send changes to face nodal points to the seconddevice, and the second device may in turn manipulate the original facemodel (which already includes image data) to give the appearance of areal-time video chat session. In this manner, a video stream may beachieved while minimizing data transferred from a first device to asecond device. Of course, it is to be appreciated that any number ofdevices may be included in the exchange of the face model andtransmittal of the real-time face model.

More illustrative information will now be set forth regarding variousoptional architectures and uses in which the foregoing method may or maynot be implemented, per the desires of the user. It should be stronglynoted that the following information is set forth for illustrativepurposes and should not be construed as limiting in any manner. Any ofthe following features may be optionally incorporated with or withoutthe exclusion of other features described.

FIG. 2A illustrates a method 200 for determining face data points, inaccordance with one embodiment. As an option, the method 200 may beimplemented in the context of any one or more of the embodiments setforth in any previous and/or subsequent figure(s) and/or descriptionthereof. Of course, however, the method 200 may be implemented in thecontext of any desired environment. Further, the aforementioneddefinitions may equally apply to the description below.

As shown, an image 201A may be received and outlines 201B of facialfeatures may be extracted. In one embodiment, such outlines 201B maytrace around facial features. Additionally, such facial features maycorrespond with face nodal points. In another embodiment, the image 201Amay be converted into a mesh (or other geometric) frame by whichcontours and features may determined and extracted. More generally, anytechnically feasible technique may be used to characterize facialfeatures and feature positions in image 201A.

After determining outlines 201B of facial features, further processing201C may result in minute data points 201D and main data points 201E. Inone embodiment, minute data points 201D may be sufficient to be sent tomanipulate the face model in real time. If additional facial data pointsare needed, then minute data points 201D may be sent as well to moreaccurately modify the face model. Of course, it is to be appreciatedthat any number of data points may be constructed (whether minute datapoints 201D or main data points 201E) and transmitted.

As an example, image 201A may be processed, resulting in outlines 201B,minute data points 201D, and main data points 201E. A package including201A and any or all of outlines 201B, minute data points 201D, and maindata points 201E may be sent to receiving device(s). Image 201A may thenbe manipulated at the receiving device(s) by adjusting any or all ofoutlines 201B, minute data points 201D, and main data points 201E byreceiving, in real-time, updated locations for any or all of outlines201B, minute data points 201D, and main data points 201E. In thismanner, any change in the face may be sent as a change in location forany of outlines 201B, minute data points 201D, and main data points201E. In one embodiment, portions of image 201A are texture-mapped ontoa geometric mesh constructed to model the face. Furthermore, movementsof data points for the geometric mesh (e.g., minute data points 201D)are tracked by the first device and transmitted to the second device,which modifies and renders the geometric mesh based on the movements.

FIG. 2B illustrates one or more parameters 203, in accordance with oneembodiment. As an option, the one or more parameters 203 may beimplemented in the context of any one or more of the embodiments setforth in any previous and/or subsequent figure(s) and/or descriptionthereof. Of course, however, the one or more parameters 203 may beimplemented in the context of any desired environment. Further, theaforementioned definitions may equally apply to the description below.

As shown, a parameter 1 202, parameter 2 204, parameter 3 206, andparameter 4 208 are included on a plot. Between time 1 and time 2,variations for each of parameters 1-4 202-208 are shown. In the contextof the present description, parameters 1-4 202-208 may relate toanimation parameters associated with a sequence of video frames. Forexample, parameters 1-4 202-208 may relate to tracking a movement of theeyes, a scrunching of the nose, a shifting of the jaw, a movement of theface bone structure, a raising or lowering of the eyebrows, a movementof the lips (e.g. smile, frown, etc.), etc. In this manner, each of theparameters 1-4 202-208 may each relate to a specific movement of theface. Of course, it is to be appreciated that additional parameters(beyond parameters 1-4 202-208) may be used to track a specific featureof the face. As such, in one embodiment, parameters 1-4 202-208represent a set of time series values for animating a face model. Inother words, a face feature is tracked with respect to time. Thetracking may be performed by the first device.

Additionally, a correlation between parameters 1-4 202-208 may be usedto associate a first parameter to a second (or any other number) ofparameters. For example, a smile parameter may also generally beassociated with a slight raise of the eyebrows, and an opening of theeyes. Such parameters may be further correlated to audio (e.g. voice)such that when a tone of voice is captured, visual parameters associatedwith the tone of voice may be captured.

In one embodiment, a user may start to smile at Time 1, causing facedata points to move relative to other facial features. One or more ofthe parameters 1-4 202-208 may quantify such movement associated withthe user smiling so that a corresponding face model may be rendered tosmile accordingly. Such parameters may be estimated and/or collected tocreate animation controls associated with the face model for rendering avideo at the second device of the face of the user at the first device.In one embodiment, such parameters may be correlated with an audio suchthat, when audio is additionally captured, the correlation between theaudio and the parameters may be used to modify the parameters (andchange the underlying image) based on the later captured audio.

In one usage mode, a first user may position a camera on a firstsmartphone (e.g., the first device) to capture video frames of theirface, and parameters 1-4 202-208 may be estimated from the video frames.For example, the video frames may be processed by a processing unitwithin the first smartphone to estimate the parameters 1-4 202-208.Parameters 1-4 202-208 are transmitted to a second smartphone (e.g., thesecond device) and used to render video frames of the face modelanimated according to the parameters 1-4 202-208. A second user may viewthe rendered video frames as a depiction of the first user talking in avideo conference call. In certain embodiments, audio may be correlatedwith expected face model movements and used to create an inference ofhow the face model of the first user should be animated to be consistentwith their spoken words. In one embodiment, such inference may be usedto manipulate the face model as discussed in relation to FIG. 2A andFIG. 1 . In one usage scenario, the first user may not reliably face thecamera of the first smartphone, and without reliably facing the camera,estimating parameters 1-4 202-208 may not be possible from video inputalone. Therefore, the parameters 1-4 202-208 may be estimated insteadfrom audio input from the first user, with phonetic deconstruction ofspeech from the first user providing a basis for estimating parameters1-4 202-208. Phonetic elements of speech and transitions betweendifferent phonetic elements have a typical mouth and/or facial positionand movement, respectively. When inferring the parameters and/or anoverall expression from audio input, the face model may be posed into anoverall neutral position, with facial movement associated with speechanimating the face model based on inferred movement from the phoneticelements. Additionally, head movement (or another parameter associatedwith the face and/or accompanying area) may be provided, such as whenlaughter is detected in the audio input.

FIG. 2C illustrates a method 205 to correlate phonetics/emotions with animage depth map, in accordance with one embodiment. As an option, themethod 205 may be implemented in the context of any one or more of theembodiments set forth in any previous and/or subsequent figure(s) and/ordescription thereof. Of course, however, the method 205 may beimplemented in the context of any desired environment. Further, theaforementioned definitions may equally apply to the description below.

As shown, the method may include an image depth map 210, an audio map212, and a correlation map 214. In one embodiment, the correlation mapmay be created based on the face model (and associated face data points)of FIG. 1 and FIG. 2A, and the audio map of FIG. 2B. In one embodiment,the image depth map 210 is used to generate a geometric model andmovement estimates of the face of the first user. Animation parameters(e.g. 1-4 202-208) derived from the image dept map 210 may betransmitted to the second device. In alternative embodiments, no depthmap is available (e.g., lack of hardware support) and image data is usedto generate the geometric model and movement estimates.

In one embodiment, the correlation map 214 may include matching up audioassociated phonetics, intonations, emotions, etc. with actual geometricdata points associated with a face. Such correlation map may beinitially created and/or captured over time (to improve accuracy).Additionally, the correlation map 214 may be sent to one or morerecipients. After receiving the correlation map and the face model (e.g.of FIG. 1 ), the second device may receive face data points thereafterto manipulate the face model. Furthermore, the first device may receiveaudio data and generate animation parameters based on the audio data;the first device may transmit the animation parameters and audio data tothe second device. In general, the audio data, in combination with thecorrelation map already received, may be used to manipulate the facemodel such that the face model may be modified in real time to correlateaudio data with visual modifications. In an alternative embodiment,rather than sending all data associated with a video conference, audiodata may be sent and used to modify a face model such that the facemodel responds in a manner consistent with how the user responds in realtime. In various embodiments, inferring facial animation based onphonetic elements may be performed at the sender side (e.g., firstsmartphone), at the receiver side (e.g., second smartphone), or at anintermediary server. Furthermore, synchronizing audio with animation ofthe face model may be performed at the sender side, the receiver side,or the intermediary server.

FIG. 2D illustrates a method 207 to manipulate a face model, inaccordance with one embodiment. As an option, the method 207 may beimplemented in the context of any one or more of the embodiments setforth in any previous and/or subsequent figure(s) and/or descriptionthereof. Of course, however, the method 207 may be implemented in thecontext of any desired environment. Further, the aforementioneddefinitions may equally apply to the description below.

As shown, a face model is received (see operation 207A), and face datapoints are received (see operation 207B). Next, the face model ismanipulated based on the received face data points. See operation 207C.As an example, an initial face model of an individual may be received.Thereafter, rather than sending the entire face model (or even the datastream of such face), face data points may be received to manipulate theface. For example, consistent with FIG. 2A, such face data points may bebased on visual modification to the face. In another embodiment,consistent with FIG. 2B, such face data points may be based on audioparameters (and thence an audio map). Alternatively, parameters 1-4202-208 are generated to concisely describe the motion of a number offace data points. In one embodiment, the parameters 1-4 202-208 may bemapped to control levers on an animation rigging of the face model.

In other embodiments, after capturing an image and depth map of aperson, a picture of the person may be displayed. From such an image, aface may be extracted and a 3D model of such face may be constructedusing polygons (or other geometric forms, mesh, etc.). From the 3Dmodel, an expression may be assumed or surmised. For example, theperson's face may include an underlying skeleton, various muscles,various chunks of fat and soft tissue, etc., and each muscle mayfunction as a lever, such that, depending on how much the muscle iscontracted, it will contribute to a different expression. In such anexample, closing eyes may be the result of muscles around the eyescontracting. As such, there are a number of parameters that can be usedto convey facial expression. In one embodiment, such parameters may beused to convey facial expression through an avatar, or some depiction ofan inanimate object.

Furthermore, video and/or audio inputs may be used to generateparameters for conveying expressions for a face model in real time. Inone embodiment, such parameters may correspond with Eigenfaces (i.e.eigenvalue(s) applied to a face to convey varying degrees of canonicalfacial expressions).

In one embodiment, transmitting the 3D model (rather than image dataassociated with real time video frames) may require significantly lessbandwidth (e.g., an amount of data usage over time) because parameters(of smaller data size) may be transmitted. In this manner, in oneembodiment, video conferences may therefore occur on less than optimalnetwork connections (as less data usage would be required or used).

As an example, an individual may attempt to establish a video conferencein conditions where poor cellular data coverage may exist, or inconditions where a cellular data rate may drop suddenly (for any numberof potential reasons). Further, an individual may be located in a verynetwork congested area (high cellular data utilization, with individualbandwidth highly constrained). In such instances (in addition to manyothers), it may be desired to maintain the video conference. Further, itmay not be desired to simply convert to an audio-only call. As such,real-time, parameter-based data points may be sent and used (based onvisual or audio input) to animate, at the receiving party device (e.g.,the second device), a face model of a sending party. In practice, theremay be two face models; one face model for each of two parties on avideo call. As such, in one embodiment, the real-time face model may beautomatically transmitted based on network latency threshold or adropped packet threshold condition.

Further, if a person has already reached a periodic limit for a highercellular speed grade, then methods as described herein would allow theperson to maintain functionality (e.g. video conference capability)while utilizing a lower bandwidth connection. In one embodiment, suchreduction in bandwidth may be based on a set percentage (e.g. 50%, 75%)of an allocated limit.

In one embodiment, the face model may be commercialized. For example,rather than using a face model associated with an individual, a facemodel associated with another individual (e.g. movie star, etc.) orcharacter (e.g. Disney(tm) character, etc.) may be purchased andutilized by a caller or receiver in a video call. In this manner, facedata points may be used to manipulate a face model other than anindividual's own face model.

Further, in another embodiment, an environment may be selected to beassociated with a face model. For example, a user could select to be ina sunny or rainy environment. Or, based on emotions detected, theenvironment may change accordingly (e.g. a sad voice may cause rainclouds to appear, etc.). In one embodiment, such environment may be anoverlay to the face model or an underlay behind the face model. In oneembodiment, the first user selects an emotion prior to establishing acall, and the emotion is conveyed in the environment (e.g., as anoverlay, underlay, or combination thereof). In another embodiment,emotional state of the first user is detected in real-time and used tomodify the environment in real-time.

In one embodiment, during a high bandwidth video conference, one or moretriggers may prompt use of the face model and face data points ratherthan real time video frames. Any bandwidth and/or network latency metricmay be used as a trigger. For example, if latency increases to a certainthreshold, or if a number of dropped packets increases to a certainthreshold, such may be used to trigger use of the face model and facedata points rather than real time video frames. This mechanism allowsfor a transition from a conventional video call to a lower-bandwidthvideo call, while maintaining continuity of the call.

FIG. 3A illustrates a digital photographic system 300, in accordancewith one embodiment. As an option, the digital photographic system 300may be implemented in the context of the details of any of the Figuresdisclosed herein. Of course, however, the digital photographic system300 may be implemented in any desired environment. Further, theaforementioned definitions may equally apply to the description below.

As shown, the digital photographic system 300 may include a processorcomplex 310 coupled to a camera module 330 via an interconnect 334. Inone embodiment, the processor complex 310 is coupled to a strobe unit336. The digital photographic system 300 may also include, withoutlimitation, a display unit 312, a set of input/output devices 314,non-volatile memory 316, volatile memory 318, a wireless unit 340, andsensor devices 342, each coupled to the processor complex 310. In oneembodiment, a power management subsystem 320 is configured to generateappropriate power supply voltages for each electrical load elementwithin the digital photographic system 300. A battery 322 may beconfigured to supply electrical energy to the power management subsystem320. The battery 322 may implement any technically feasible energystorage system, including primary or rechargeable battery technologies.Of course, in other embodiments, additional or fewer features, units,devices, sensors, or subsystems may be included in the system.

In one embodiment, a strobe unit 336 may be integrated into the digitalphotographic system 300 and configured to provide strobe illumination350 during an image sample event performed by the digital photographicsystem 300. In another embodiment, a strobe unit 336 may be implementedas an independent device from the digital photographic system 300 andconfigured to provide strobe illumination 350 during an image sampleevent performed by the digital photographic system 300. The strobe unit336 may comprise one or more LED devices, a gas-discharge illuminator(e.g. a Xenon strobe device, a Xenon flash lamp, etc.), or any othertechnically feasible illumination device. In certain embodiments, two ormore strobe units are configured to synchronously generate strobeillumination in conjunction with sampling an image. In one embodiment,the strobe unit 336 is controlled through a strobe control signal 338 toeither emit the strobe illumination 350 or not emit the strobeillumination 350. The strobe control signal 338 may be implemented usingany technically feasible signal transmission protocol. The strobecontrol signal 338 may indicate a strobe parameter (e.g. strobeintensity, strobe color, strobe time, etc.), for directing the strobeunit 336 to generate a specified intensity and/or color of the strobeillumination 350. The strobe control signal 338 may be generated by theprocessor complex 310, the camera module 330, or by any othertechnically feasible combination thereof. In one embodiment, the strobecontrol signal 338 is generated by a camera interface unit within theprocessor complex 310 and transmitted to both the strobe unit 336 andthe camera module 330 via the interconnect 334. In another embodiment,the strobe control signal 338 is generated by the camera module 330 andtransmitted to the strobe unit 336 via the interconnect 334.

Optical scene information 352, which may include at least a portion ofthe strobe illumination 350 reflected from objects in the photographicscene, is focused as an optical image onto an image sensor 332 withinthe camera module 330. The image sensor 332 generates an electronicrepresentation of the optical image. The electronic representationcomprises spatial color intensity information, which may includedifferent color intensity samples (e.g. red, green, and blue light,etc.). In other embodiments, the spatial color intensity information mayalso include samples for white light. The electronic representation istransmitted to the processor complex 310 via the interconnect 334, whichmay implement any technically feasible signal transmission protocol.

In one embodiment, input/output devices 314 may include, withoutlimitation, a capacitive touch input surface, a resistive tablet inputsurface, one or more buttons, one or more knobs, light-emitting devices,light detecting devices, sound emitting devices, sound detectingdevices, or any other technically feasible device for receiving userinput and converting the input to electrical signals, or convertingelectrical signals into a physical signal. In one embodiment, theinput/output devices 314 include a capacitive touch input surfacecoupled to a display unit 312. A touch entry display system may includethe display unit 312 and a capacitive touch input surface, also coupledto processor complex 310.

Additionally, in other embodiments, non-volatile (NV) memory 316 isconfigured to store data when power is interrupted. In one embodiment,the NV memory 316 comprises one or more flash memory devices (e.g. ROM,PCM, FeRAM, FRAM, PRAM, MRAM, NRAM, etc.). The NV memory 316 comprises anon-transitory computer-readable medium, which may be configured toinclude programming instructions for execution by one or more processingunits within the processor complex 310. The programming instructions mayimplement, without limitation, an operating system (OS), UI softwaremodules, image processing and storage software modules, one or moreinput/output devices 314 connected to the processor complex 310, one ormore software modules for sampling an image stack through camera module330, one or more software modules for presenting the image stack or oneor more synthetic images generated from the image stack through thedisplay unit 312. As an example, in one embodiment, the programminginstructions may also implement one or more software modules for mergingimages or portions of images within the image stack, aligning at leastportions of each image within the image stack, or a combination thereof.In another embodiment, the processor complex 310 may be configured toexecute the programming instructions, which may implement one or moresoftware modules operable to create a high dynamic range (HDR) image.

Still yet, in one embodiment, one or more memory devices comprising theNV memory 316 may be packaged as a module configured to be installed orremoved by a user. In one embodiment, volatile memory 318 comprisesdynamic random access memory (DRAM) configured to temporarily storeprogramming instructions, image data such as data associated with animage stack, and the like, accessed during the course of normaloperation of the digital photographic system 300. Of course, thevolatile memory may be used in any manner and in association with anyother input/output device 314 or sensor device 342 attached to theprocess complex 310.

In one embodiment, sensor devices 342 may include, without limitation,one or more of an accelerometer to detect motion and/or orientation, anelectronic gyroscope to detect motion and/or orientation, a magneticflux detector to detect orientation, a global positioning system (GPS)module to detect geographic position, or any combination thereof. Ofcourse, other sensors, including but not limited to a motion detectionsensor, a proximity sensor, an RGB light sensor, a gesture sensor, a 3-Dinput image sensor, a pressure sensor, and an indoor position sensor,may be integrated as sensor devices. In one embodiment, the sensordevices may be one example of input/output devices 314.

Wireless unit 340 may include one or more digital radios configured tosend and receive digital data. In particular, the wireless unit 340 mayimplement wireless standards (e.g. WiFi, Bluetooth, NFC, etc.), and mayimplement digital cellular telephony standards for data communication(e.g. CDMA, 3G, 4G, LTE, LTE-Advanced, etc.). Of course, any wirelessstandard or digital cellular telephony standards may be used.

In one embodiment, the digital photographic system 300 is configured totransmit one or more digital photographs to a network-based (online) or“cloud-based” photographic media service via the wireless unit 340. Theone or more digital photographs may reside within either the NV memory316 or the volatile memory 318, or any other memory device associatedwith the processor complex 310. In one embodiment, a user may possesscredentials to access an online photographic media service and totransmit one or more digital photographs for storage to, retrieval from,and presentation by the online photographic media service. Thecredentials may be stored or generated within the digital photographicsystem 300 prior to transmission of the digital photographs. The onlinephotographic media service may comprise a social networking service,photograph sharing service, or any other network-based service thatprovides storage of digital photographs, processing of digitalphotographs, transmission of digital photographs, sharing of digitalphotographs, or any combination thereof. In certain embodiments, one ormore digital photographs are generated by the online photographic mediaservice based on image data (e.g. image stack, HDR image stack, imagepackage, etc.) transmitted to servers associated with the onlinephotographic media service. In such embodiments, a user may upload oneor more source images from the digital photographic system 300 forprocessing by the online photographic media service.

In one embodiment, the digital photographic system 300 comprises atleast one instance of a camera module 330. In another embodiment, thedigital photographic system 300 comprises a plurality of camera modules330. Such an embodiment may also include at least one strobe unit 336configured to illuminate a photographic scene, sampled as multiple viewsby the plurality of camera modules 330. The plurality of camera modules330 may be configured to sample a wide angle view (e.g., greater thanforty-five degrees of sweep among cameras) to generate a panoramicphotograph. In one embodiment, a plurality of camera modules 330 may beconfigured to sample two or more narrow angle views (e.g., less thanforty-five degrees of sweep among cameras) to generate a stereoscopicphotograph. In other embodiments, a plurality of camera modules 330 maybe configured to generate a 3-D image or to otherwise display a depthperspective (e.g. a z-component, etc.) as shown on the display unit 312or any other display device.

In one embodiment, a display unit 312 may be configured to display atwo-dimensional array of pixels to form an image for display. Thedisplay unit 312 may comprise a liquid-crystal (LCD) display, alight-emitting diode (LED) display, an organic LED display, or any othertechnically feasible type of display. In certain embodiments, thedisplay unit 312 may be able to display a narrower dynamic range ofimage intensity values than a complete range of intensity values sampledfrom a photographic scene, such as within a single HDR image or over aset of two or more images comprising a multiple exposure or HDR imagestack. In one embodiment, images comprising an image stack may be mergedaccording to any technically feasible HDR blending technique to generatea synthetic image for display within dynamic range constraints of thedisplay unit 312. In one embodiment, the limited dynamic range mayspecify an eight-bit per color channel binary representation ofcorresponding color intensities. In other embodiments, the limiteddynamic range may specify more than eight-bits (e.g., 10 bits, 12 bits,or 14 bits, etc.) per color channel binary representation.

FIG. 3B illustrates a processor complex 310 within the digitalphotographic system 300 of FIG. 3A, in accordance with one embodiment.As an option, the processor complex 310 may be implemented in thecontext of the details of any of the Figures disclosed herein. Ofcourse, however, the processor complex 310 may be implemented in anydesired environment. Further, the aforementioned definitions may equallyapply to the description below.

As shown, the processor complex 310 includes a processor subsystem 360and may include a memory subsystem 362. In one embodiment, processorcomplex 310 may comprise a system on a chip (SoC) device that implementsprocessor subsystem 360, and memory subsystem 362 comprises one or moreDRAM devices coupled to the processor subsystem 360. In anotherembodiment, the processor complex 310 may comprise a multi-chip module(MCM) encapsulating the SoC device and the one or more DRAM devicescomprising the memory subsystem 362.

The processor subsystem 360 may include, without limitation, one or morecentral processing unit (CPU) cores 370, a memory interface 380,input/output interfaces unit 384, and a display interface unit 382, eachcoupled to an interconnect 374. The one or more CPU cores 370 may beconfigured to execute instructions residing within the memory subsystem362, volatile memory 318, NV memory 316, or any combination thereof.Each of the one or more CPU cores 370 may be configured to retrieve andstore data through interconnect 374 and the memory interface 380. In oneembodiment, each of the one or more CPU cores 370 may include a datacache, and an instruction cache. Additionally, two or more of the CPUcores 370 may share a data cache, an instruction cache, or anycombination thereof. In one embodiment, a cache hierarchy is implementedto provide each CPU core 370 with a private cache layer, and a sharedcache layer.

In some embodiments, processor subsystem 360 may include one or moregraphics processing unit (GPU) cores 372. Each GPU core 372 may comprisea plurality of multi-threaded execution units that may be programmed toimplement, without limitation, graphics acceleration functions. Invarious embodiments, the GPU cores 372 may be configured to executemultiple thread programs according to well-known standards (e.g.OpenGL™, WebGL™, OpenCL™, CUDA™, etc.), and/or any other programmablerendering graphic standard. In certain embodiments, at least one GPUcore 372 implements at least a portion of a motion estimation function,such as a well-known Harris detector or a well-known Hessian-Laplacedetector. Such a motion estimation function may be used at least in partto align images or portions of images within an image stack. Forexample, in one embodiment, an HDR image may be compiled based on animage stack, where two or more images are first aligned prior tocompiling the HDR image.

As shown, the interconnect 374 is configured to transmit data betweenand among the memory interface 380, the display interface unit 382, theinput/output interfaces unit 384, the CPU cores 370, and the GPU cores372. In various embodiments, the interconnect 374 may implement one ormore buses, one or more rings, a cross-bar, a mesh, or any othertechnically feasible data transmission structure or technique. Thememory interface 380 is configured to couple the memory subsystem 362 tothe interconnect 374. The memory interface 380 may also couple NV memory316, volatile memory 318, or any combination thereof to the interconnect374. The display interface unit 382 may be configured to couple adisplay unit 312 to the interconnect 374. The display interface unit 382may implement certain frame buffer functions (e.g. frame refresh, etc.).Alternatively, in another embodiment, the display unit 312 may implementcertain frame buffer functions (e.g. frame refresh, etc.). Theinput/output interfaces unit 384 may be configured to couple variousinput/output devices to the interconnect 374.

In certain embodiments, a camera module 330 is configured to storeexposure parameters for sampling each image associated with an imagestack. For example, in one embodiment, when directed to sample aphotographic scene, the camera module 330 may sample a set of imagescomprising the image stack according to stored exposure parameters. Asoftware module comprising programming instructions executing within aprocessor complex 310 may generate and store the exposure parametersprior to directing the camera module 330 to sample the image stack. Inother embodiments, the camera module 330 may be used to meter an imageor an image stack, and the software module comprising programminginstructions executing within a processor complex 310 may generate andstore metering parameters prior to directing the camera module 330 tocapture the image. Of course, the camera module 330 may be used in anymanner in combination with the processor complex 310.

In one embodiment, exposure parameters associated with images comprisingthe image stack may be stored within an exposure parameter datastructure that includes exposure parameters for one or more images. Inanother embodiment, a camera interface unit (not shown in FIG. 3B)within the processor complex 310 may be configured to read exposureparameters from the exposure parameter data structure and to transmitassociated exposure parameters to the camera module 330 in preparationof sampling a photographic scene. After the camera module 330 isconfigured according to the exposure parameters, the camera interfacemay direct the camera module 330 to sample the photographic scene; thecamera module 330 may then generate a corresponding image stack. Theexposure parameter data structure may be stored within the camerainterface unit, a memory circuit within the processor complex 310,volatile memory 318, NV memory 316, the camera module 330, or within anyother technically feasible memory circuit. Further, in anotherembodiment, a software module executing within processor complex 310 maygenerate and store the exposure parameter data structure.

FIG. 3C illustrates a digital camera 302, in accordance with oneembodiment. As an option, the digital camera 302 may be implemented inthe context of the details of any of the Figures disclosed herein. Ofcourse, however, the digital camera 302 may be implemented in anydesired environment. Further, the aforementioned definitions may equallyapply to the description below.

In one embodiment, the digital camera 302 may be configured to include adigital photographic system, such as digital photographic system 300 ofFIG. 3A. As shown, the digital camera 302 includes a camera module 330,which may include optical elements configured to focus optical sceneinformation representing a photographic scene onto an image sensor,which may be configured to convert the optical scene information to anelectronic representation of the photographic scene.

Additionally, the digital camera 302 may include a strobe unit 336, andmay include a shutter release button 315 for triggering a photographicsample event, whereby digital camera 302 samples one or more imagescomprising the electronic representation. In other embodiments, anyother technically feasible shutter release mechanism may trigger thephotographic sample event (e.g. such as a timer trigger or remotecontrol trigger, etc.).

FIG. 3D illustrates a wireless mobile device 376, in accordance with oneembodiment. As an option, the mobile device 376 may be implemented inthe context of the details of any of the Figures disclosed herein. Ofcourse, however, the mobile device 376 may be implemented in any desiredenvironment. Further, the aforementioned definitions may equally applyto the description below.

In one embodiment, the mobile device 376 may be configured to include adigital photographic system (e.g. such as digital photographic system300 of FIG. 3A), which is configured to sample a photographic scene. Invarious embodiments, a camera module 330 may include optical elementsconfigured to focus optical scene information representing thephotographic scene onto an image sensor, which may be configured toconvert the optical scene information to an electronic representation ofthe photographic scene. Further, a shutter release command may begenerated through any technically feasible mechanism, such as a virtualbutton, which may be activated by a touch gesture on a touch entrydisplay system comprising display unit 312, or a physical button, whichmay be located on any face or surface of the mobile device 376. Ofcourse, in other embodiments, any number of other buttons, externalinputs/outputs, or digital inputs/outputs may be included on the mobiledevice 376, and which may be used in conjunction with the camera module330.

As shown, in one embodiment, a touch entry display system comprisingdisplay unit 312 is disposed on the opposite side of mobile device 376from camera module 330. In certain embodiments, the mobile device 376includes a user-facing camera module 331 and may include a user-facingstrobe unit (not shown). Of course, in other embodiments, the mobiledevice 376 may include any number of user-facing camera modules orrear-facing camera modules, as well as any number of user-facing strobeunits or rear-facing strobe units.

In some embodiments, the digital camera 302 and the mobile device 376may each generate and store a synthetic image based on an image stacksampled by camera module 330. The image stack may include one or moreimages sampled under ambient lighting conditions, one or more imagessampled under strobe illumination from strobe unit 336, or a combinationthereof.

FIG. 3E illustrates camera module 330, in accordance with oneembodiment. As an option, the camera module 330 may be implemented inthe context of the details of any of the Figures disclosed herein. Ofcourse, however, the camera module 330 may be implemented in any desiredenvironment. Further, the aforementioned definitions may equally applyto the description below.

In one embodiment, the camera module 330 may be configured to controlstrobe unit 336 through strobe control signal 338. As shown, a lens 390is configured to focus optical scene information 352 onto image sensor332 to be sampled. In one embodiment, image sensor 332 advantageouslycontrols detailed timing of the strobe unit 336 though the strobecontrol signal 338 to reduce inter-sample time between an image sampledwith the strobe unit 336 enabled, and an image sampled with the strobeunit 336 disabled. For example, the image sensor 332 may enable thestrobe unit 336 to emit strobe illumination 350 less than onemicrosecond (or any desired length) after image sensor 332 completes anexposure time associated with sampling an ambient image and prior tosampling a strobe image.

In other embodiments, the strobe illumination 350 may be configuredbased on a desired one or more target points. For example, in oneembodiment, the strobe illumination 350 may light up an object in theforeground, and depending on the length of exposure time, may also lightup an object in the background of the image. In one embodiment, once thestrobe unit 336 is enabled, the image sensor 332 may then immediatelybegin exposing a strobe image. The image sensor 332 may thus be able todirectly control sampling operations, including enabling and disablingthe strobe unit 336 associated with generating an image stack, which maycomprise at least one image sampled with the strobe unit 336 disabled,and at least one image sampled with the strobe unit 336 either enabledor disabled. In one embodiment, data comprising the image stack sampledby the image sensor 332 is transmitted via interconnect 334 to a camerainterface unit 386 within processor complex 310. In some embodiments,the camera module 330 may include an image sensor controller (e.g.,controller 333 of FIG. 3G), which may be configured to generate thestrobe control signal 338 in conjunction with controlling operation ofthe image sensor 332.

FIG. 3F illustrates a camera module 330, in accordance with oneembodiment. As an option, the camera module 330 may be implemented inthe context of the details of any of the Figures disclosed herein. Ofcourse, however, the camera module 330 may be implemented in any desiredenvironment. Further, the aforementioned definitions may equally applyto the description below.

In one embodiment, the camera module 330 may be configured to sample animage based on state information for strobe unit 336. The stateinformation may include, without limitation, one or more strobeparameters (e.g. strobe intensity, strobe color, strobe time, etc.), fordirecting the strobe unit 336 to generate a specified intensity and/orcolor of the strobe illumination 350. In one embodiment, commands forconfiguring the state information associated with the strobe unit 336may be transmitted through a strobe control signal 338, which may bemonitored by the camera module 330 to detect when the strobe unit 336 isenabled. For example, in one embodiment, the camera module 330 maydetect when the strobe unit 336 is enabled or disabled within amicrosecond or less of the strobe unit 336 being enabled or disabled bythe strobe control signal 338. To sample an image requiring strobeillumination, a camera interface unit 386 may enable the strobe unit 336by sending an enable command through the strobe control signal 338. Inone embodiment, the camera interface unit 386 may be included as aninterface of input/output interfaces 384 in a processor subsystem 360 ofthe processor complex 310 of FIG. 3B. The enable command may comprise asignal level transition, a data packet, a register write, or any othertechnically feasible transmission of a command. The camera module 330may sense that the strobe unit 336 is enabled and then cause imagesensor 332 to sample one or more images requiring strobe illuminationwhile the strobe unit 336 is enabled. In such an implementation, theimage sensor 332 may be configured to wait for an enable signal destinedfor the strobe unit 336 as a trigger signal to begin sampling a newexposure.

In one embodiment, camera interface unit 386 may transmit exposureparameters and commands to camera module 330 through interconnect 334.In certain embodiments, the camera interface unit 386 may be configuredto directly control strobe unit 336 by transmitting control commands tothe strobe unit 336 through strobe control signal 338. By directlycontrolling both the camera module 330 and the strobe unit 336, thecamera interface unit 386 may cause the camera module 330 and the strobeunit 336 to perform their respective operations in precise timesynchronization. In one embodiment, precise time synchronization may beless than five hundred microseconds of event timing error. Additionally,event timing error may be a difference in time from an intended eventoccurrence to the time of a corresponding actual event occurrence.

In another embodiment, camera interface unit 386 may be configured toaccumulate statistics while receiving image data from camera module 330.In particular, the camera interface unit 386 may accumulate exposurestatistics for a given image while receiving image data for the imagethrough interconnect 334. Exposure statistics may include, withoutlimitation, one or more of an intensity histogram, a count ofover-exposed pixels, a count of under-exposed pixels, anintensity-weighted sum of pixel intensity, or any combination thereof.The camera interface unit 386 may present the exposure statistics asmemory-mapped storage locations within a physical or virtual addressspace defined by a processor, such as one or more of CPU cores 370,within processor complex 310. In one embodiment, exposure statisticsreside in storage circuits that are mapped into a memory-mapped registerspace, which may be accessed through the interconnect 334. In otherembodiments, the exposure statistics are transmitted in conjunction withtransmitting pixel data for a captured image. For example, the exposurestatistics for a given image may be transmitted as in-line data,following transmission of pixel intensity data for the captured image.Exposure statistics may be calculated, stored, or cached within thecamera interface unit 386. In other embodiments, an image sensorcontroller within camera module 330 may be configured to accumulate theexposure statistics and transmit the exposure statistics to processorcomplex 310, such as by way of camera interface unit 386. In oneembodiment, the exposure statistics are accumulated within the cameramodule 330 and transmitted to the camera interface unit 386, either inconjunction with transmitting image data to the camera interface unit386, or separately from transmitting image data.

In one embodiment, camera interface unit 386 may accumulate colorstatistics for estimating scene white-balance. Any technically feasiblecolor statistics may be accumulated for estimating white balance, suchas a sum of intensities for different color channels comprising red,green, and blue color channels. The sum of color channel intensities maythen be used to perform a white-balance color correction on anassociated image, according to a white-balance model such as agray-world white-balance model. In other embodiments, curve-fittingstatistics are accumulated for a linear or a quadratic curve fit usedfor implementing white-balance correction on an image. As with theexposure statistics, the color statistics may be presented asmemory-mapped storage locations within processor complex 310. In oneembodiment, the color statistics may be mapped in a memory-mappedregister space, which may be accessed through interconnect 334. In otherembodiments, the color statistics may be transmitted in conjunction withtransmitting pixel data for a captured image. For example, in oneembodiment, the color statistics for a given image may be transmitted asin-line data, following transmission of pixel intensity data for theimage. Color statistics may be calculated, stored, or cached within thecamera interface 386. In other embodiments, the image sensor controllerwithin camera module 330 may be configured to accumulate the colorstatistics and transmit the color statistics to processor complex 310,such as by way of camera interface unit 386. In one embodiment, thecolor statistics may be accumulated within the camera module 330 andtransmitted to the camera interface unit 386, either in conjunction withtransmitting image data to the camera interface unit 386, or separatelyfrom transmitting image data.

In one embodiment, camera interface unit 386 may accumulate spatialcolor statistics for performing color-matching between or among images,such as between or among an ambient image and one or more images sampledwith strobe illumination. As with the exposure statistics, the spatialcolor statistics may be presented as memory-mapped storage locationswithin processor complex 310. In one embodiment, the spatial colorstatistics are mapped in a memory-mapped register space. In anotherembodiment the camera module may be configured to accumulate the spatialcolor statistics, which may be accessed through interconnect 334. Inother embodiments, the color statistics may be transmitted inconjunction with transmitting pixel data for a captured image. Forexample, in one embodiment, the color statistics for a given image maybe transmitted as in-line data, following transmission of pixelintensity data for the image. Color statistics may be calculated,stored, or cached within the camera interface 386.

In one embodiment, camera module 330 may transmit strobe control signal338 to strobe unit 336, enabling the strobe unit 336 to generateillumination while the camera module 330 is sampling an image. Inanother embodiment, camera module 330 may sample an image illuminated bystrobe unit 336 upon receiving an indication signal from camerainterface unit 386 that the strobe unit 336 is enabled. In yet anotherembodiment, camera module 330 may sample an image illuminated by strobeunit 336 upon detecting strobe illumination within a photographic scenevia a rapid rise in scene illumination. In one embodiment, a rapid risein scene illumination may include at least a rate of increasingintensity consistent with that of enabling strobe unit 336. In still yetanother embodiment, camera module 330 may enable strobe unit 336 togenerate strobe illumination while sampling one image, and disable thestrobe unit 336 while sampling a different image.

FIG. 3G illustrates camera module 330, in accordance with oneembodiment. As an option, the camera module 330 may be implemented inthe context of the details of any of the Figures disclosed herein. Ofcourse, however, the camera module 330 may be implemented in any desiredenvironment. Further, the aforementioned definitions may equally applyto the description below.

In one embodiment, the camera module 330 may be in communication with anapplication processor 335. The camera module 330 is shown to includeimage sensor 332 in communication with a controller 333. Further, thecontroller 333 is shown to be in communication with the applicationprocessor 335.

In one embodiment, the application processor 335 may reside outside ofthe camera module 330. As shown, the lens 390 may be configured to focusoptical scene information to be sampled onto image sensor 332. Theoptical scene information sampled by the image sensor 332 may then becommunicated from the image sensor 332 to the controller 333 for atleast one of subsequent processing and communication to the applicationprocessor 335. In another embodiment, the controller 333 may controlstorage of the optical scene information sampled by the image sensor332, or storage of processed optical scene information.

In another embodiment, the controller 333 may enable a strobe unit toemit strobe illumination for a short time duration (e.g. less than tenmilliseconds) after image sensor 332 completes an exposure timeassociated with sampling an ambient image. Further, the controller 333may be configured to generate strobe control signal 338 in conjunctionwith controlling operation of the image sensor 332.

In one embodiment, the image sensor 332 may be a complementary metaloxide semiconductor (CMOS) sensor or a charge-coupled device (CCD)sensor. In another embodiment, the controller 333 and the image sensor332 may be packaged together as an integrated system, multi-chip module,multi-chip stack, or integrated circuit. In yet another embodiment, thecontroller 333 and the image sensor 332 may comprise discrete packages.In one embodiment, the controller 333 may provide circuitry forreceiving optical scene information from the image sensor 332,processing of the optical scene information, timing of variousfunctionalities, and signaling associated with the application processor335. Further, in another embodiment, the controller 333 may providecircuitry for control of one or more of exposure, shuttering, whitebalance, and gain adjustment. Processing of the optical sceneinformation by the circuitry of the controller 333 may include one ormore of gain application, amplification, and analog-to-digitalconversion. After processing the optical scene information, thecontroller 333 may transmit corresponding digital pixel data, such as tothe application processor 335.

In one embodiment, the application processor 335 may be implemented onprocessor complex 310 and at least one of volatile memory 318 and NVmemory 316, or any other memory device and/or system. The applicationprocessor 335 may be previously configured for processing of receivedoptical scene information or digital pixel data communicated from thecamera module 330 to the application processor 335.

FIG. 4 illustrates a network service system 400, in accordance with oneembodiment. As an option, the network service system 400 may beimplemented in the context of the details of any of the Figuresdisclosed herein. Of course, however, the network service system 400 maybe implemented in any desired environment. Further, the aforementioneddefinitions may equally apply to the description below.

In one embodiment, the network service system 400 may be configured toprovide network access to a device implementing a digital photographicsystem. As shown, network service system 400 includes a wireless mobiledevice 376, a wireless access point 472, a data network 474, a datacenter 480, and a data center 481. The wireless mobile device 376 maycommunicate with the wireless access point 472 via a digital radio link471 to send and receive digital data, including data associated withdigital images. The wireless mobile device 376 and the wireless accesspoint 472 may implement any technically feasible transmission techniquesfor transmitting digital data via digital radio link 471 withoutdeparting the scope and spirit of the present invention. In certainembodiments, one or more of data centers 480, 481 may be implementedusing virtual constructs so that each system and subsystem within agiven data center 480, 481 may comprise virtual machines configured toperform data processing and network data transmission tasks. In otherimplementations, one or more of data centers 480, 481 may be physicallydistributed over a plurality of physical sites.

The wireless mobile device 376 may comprise a smart phone configured toinclude a digital camera, a digital camera configured to includewireless network connectivity, a reality augmentation device, a laptopconfigured to include a digital camera and wireless networkconnectivity, or any other technically feasible computing deviceconfigured to include a digital photographic system and wireless networkconnectivity.

In various embodiments, the wireless access point 472 may be configuredto communicate with wireless mobile device 376 via the digital radiolink 471 and to communicate with the data network 474 via anytechnically feasible transmission media, such as any electrical,optical, or radio transmission media. For example, in one embodiment,wireless access point 472 may communicate with data network 474 throughan optical fiber coupled to the wireless access point 472 and to arouter system or a switch system within the data network 474. A networklink 475, such as a wide area network (WAN) link, may be configured totransmit data between the data network 474 and the data center 480.

In one embodiment, the data network 474 may include routers, switches,long-haul transmission systems, provisioning systems, authorizationsystems, and any technically feasible combination of communications andoperations subsystems configured to convey data between networkendpoints, such as between the wireless access point 472 and the datacenter 480. In one implementation scenario, wireless mobile device 376may comprise one of a plurality of wireless mobile devices configured tocommunicate with the data center 480 via one or more wireless accesspoints coupled to the data network 474.

Additionally, in various embodiments, the data center 480 may include,without limitation, a switch/router 482 and at least one data servicesystem 484. The switch/router 482 may be configured to forward datatraffic between and among a network link 475, and each data servicesystem 484. The switch/router 482 may implement any technically feasibletransmission techniques, such as Ethernet media layer transmission,layer 2 switching, layer 3 routing, and the like. The switch/router 482may comprise one or more individual systems configured to transmit databetween the data service systems 484 and the data network 474.

In one embodiment, the switch/router 482 may implement session-levelload balancing among a plurality of data service systems 484. Each dataservice system 484 may include at least one computation system 488 andmay also include one or more storage systems 486. Each computationsystem 488 may comprise one or more processing units, such as a centralprocessing unit, a graphics processing unit, or any combination thereof.A given data service system 484 may be implemented as a physical systemcomprising one or more physically distinct systems configured to operatetogether. Alternatively, a given data service system 484 may beimplemented as a virtual system comprising one or more virtual systemsexecuting on an arbitrary physical system. In certain scenarios, thedata network 474 may be configured to transmit data between the datacenter 480 and another data center 481, such as through a network link476.

In another embodiment, the network service system 400 may include anynetworked mobile devices configured to implement one or more embodimentsof the present invention. For example, in some embodiments, apeer-to-peer network, such as an ad-hoc wireless network, may beestablished between two different wireless mobile devices. In suchembodiments, digital image data may be transmitted between the twowireless mobile devices without having to send the digital image data toa data center 480.

FIG. 5 illustrates a network architecture 500, in accordance with onepossible embodiment. As shown, at least one network 502 is provided. Inthe context of the present network architecture 500, the network 502 maytake any form including, but not limited to a telecommunicationsnetwork, a local area network (LAN), a wireless network, a wide areanetwork (WAN) such as the Internet, peer-to-peer network, cable network,etc. While only one network is shown, it should be understood that twoor more similar or different networks 502 may be provided.

Coupled to the network 502 is a plurality of devices. For example, aserver computer 512 and an end user computer 508 may be coupled to thenetwork 502 for communication purposes. Such end user computer 508 mayinclude a desktop computer, lap-top computer, and/or any other type oflogic. Still yet, various other devices may be coupled to the network502 including a personal digital assistant (PDA) device 510, a mobilephone device 506, a television 504, a camera 514, etc.

FIG. 6 illustrates an exemplary system 600, in accordance with oneembodiment. As an option, the system 600 may be implemented in thecontext of any of the devices of the network architecture 500 of FIG. 5. Of course, the system 600 may be implemented in any desiredenvironment.

As shown, a system 600 is provided including at least one centralprocessor 602 which is connected to a communication bus 612. The system600 also includes main memory 604 [e.g. random access memory (RAM),etc.]. The system 600 also includes a graphics processor 608 and adisplay 610.

The system 600 may also include a secondary storage 606. The secondarystorage 606 includes, for example, a hard disk drive and/or a removablestorage drive, representing a floppy disk drive, a magnetic tape drive,a compact disk drive, etc. The removable storage drive reads from and/orwrites to a removable storage unit in a well known manner.

Computer programs, or computer control logic algorithms, may be storedin the main memory 604, the secondary storage 606, and/or any othermemory, for that matter. Such computer programs, when executed, enablethe system 600 to perform various functions (as set forth above, forexample). Memory 604, storage 606 and/or any other storage are possibleexamples of non-transitory computer-readable media.

It is noted that the techniques described herein, in an aspect, areembodied in executable instructions stored in a computer readable mediumfor use by or in connection with an instruction execution machine,apparatus, or device, such as a computer-based or processor-containingmachine, apparatus, or device. It will be appreciated by those skilledin the art that for some embodiments, other types of computer readablemedia are included which may store data that is accessible by acomputer, such as magnetic cassettes, flash memory cards, digital videodisks, Bernoulli cartridges, random access memory (RAM), read-onlymemory (ROM), and the like.

As used here, a “computer-readable medium” includes one or more of anysuitable media for storing the executable instructions of a computerprogram such that the instruction execution machine, system, apparatus,or device may read (or fetch) the instructions from the computerreadable medium and execute the instructions for carrying out thedescribed methods. Suitable storage formats include one or more of anelectronic, magnetic, optical, and electromagnetic format. Anon-exhaustive list of conventional exemplary computer readable mediumincludes: a portable computer diskette; a RAM; a ROM; an erasableprogrammable read only memory (EPROM or flash memory); optical storagedevices, including a portable compact disc (CD), a portable digitalvideo disc (DVD), a high definition DVD (HD-DVD™), a BLU-RAY disc; andthe like.

It should be understood that the arrangement of components illustratedin the Figures described are exemplary and that other arrangements arepossible. It should also be understood that the various systemcomponents (and means) defined by the claims, described below, andillustrated in the various block diagrams represent logical componentsin some systems configured according to the subject matter disclosedherein.

For example, one or more of these system components (and means) may berealized, in whole or in part, by at least some of the componentsillustrated in the arrangements illustrated in the described Figures. Inaddition, while at least one of these components are implemented atleast partially as an electronic hardware component, and thereforeconstitutes a machine, the other components may be implemented insoftware that when included in an execution environment constitutes amachine, hardware, or a combination of software and hardware.

More particularly, at least one component defined by the claims isimplemented at least partially as an electronic hardware component, suchas an instruction execution machine (e.g., a processor-based orprocessor-containing machine) and/or as specialized circuits orcircuitry (e.g., discreet logic gates interconnected to perform aspecialized function). Other components may be implemented in software,hardware, or a combination of software and hardware. Moreover, some orall of these other components may be combined, some may be omittedaltogether, and additional components may be added while still achievingthe functionality described herein. Thus, the subject matter describedherein may be embodied in many different variations, and all suchvariations are contemplated to be within the scope of what is claimed.

In the description above, the subject matter is described with referenceto acts and symbolic representations of operations that are performed byone or more devices, unless indicated otherwise. As such, it will beunderstood that such acts and operations, which are at times referred toas being computer-executed, include the manipulation by the processor ofdata in a structured form. This manipulation transforms the data ormaintains it at locations in the memory system of the computer, whichreconfigures or otherwise alters the operation of the device in a mannerwell understood by those skilled in the art. The data is maintained atphysical locations of the memory as data structures that have particularproperties defined by the format of the data. However, while the subjectmatter is being described in the foregoing context, it is not meant tobe limiting as those of skill in the art will appreciate that various ofthe acts and operations described hereinafter may also be implemented inhardware.

To facilitate an understanding of the subject matter described herein,many aspects are described in terms of sequences of actions. At leastone of these aspects defined by the claims is performed by an electronichardware component. For example, it will be recognized that the variousactions may be performed by specialized circuits or circuitry, byprogram instructions being executed by one or more processors, or by acombination of both. The description herein of any sequence of actionsis not intended to imply that the specific order described forperforming that sequence must be followed. All methods described hereinmay be performed in any suitable order unless otherwise indicated hereinor otherwise clearly contradicted by context.

The use of the terms “a” and “an” and “the” and similar referents in thecontext of describing the subject matter (particularly in the context ofthe following claims) are to be construed to cover both the singular andthe plural, unless otherwise indicated herein or clearly contradicted bycontext. Recitation of ranges of values herein are merely intended toserve as a shorthand method of referring individually to each separatevalue falling within the range, unless otherwise indicated herein, andeach separate value is incorporated into the specification as if it wereindividually recited herein. Furthermore, the foregoing description isfor the purpose of illustration only, and not for the purpose oflimitation, as the scope of protection sought is defined by the claimsas set forth hereinafter together with any equivalents thereof entitledto. The use of any and all examples, or exemplary language (e.g., “suchas”) provided herein, is intended merely to better illustrate thesubject matter and does not pose a limitation on the scope of thesubject matter unless otherwise claimed. The use of the term “based on”and other like phrases indicating a condition for bringing about aresult, both in the claims and in the written description, is notintended to foreclose any other conditions that bring about that result.No language in the specification should be construed as indicating anynon-claimed element as essential to the practice of the invention asclaimed.

The embodiments described herein included the one or more modes known tothe inventor for carrying out the claimed subject matter. Of course,variations of those embodiments will become apparent to those ofordinary skill in the art upon reading the foregoing description. Theinventor expects skilled artisans to employ such variations asappropriate, and the inventor intends for the claimed subject matter tobe practiced otherwise than as specifically described herein.Accordingly, this claimed subject matter includes all modifications andequivalents of the subject matter recited in the claims appended heretoas permitted by applicable law. Moreover, any combination of theabove-described elements in all possible variations thereof isencompassed unless otherwise indicated herein or otherwise clearlycontradicted by context.

What is claimed is:
 1. A device, comprising: a non-transitory memorystoring instructions; and one or more processors in communication withthe non-transitory memory, wherein the one or more processors executethe instructions to: receive a real-time face model, wherein thereal-time face model includes one or more face nodal points and aplurality of face parameters; receive real-time face nodal points,including additional one or more face nodal points; and manipulate thereal-time face model based on the real-time face nodal points; whereinthe device is operable such that the plurality of face parameters iscorrelated and wherein the correlation occurs synchronously; wherein thereal-time face model includes an image depth map, an audio map, and acorrelation map.
 2. The device of claim 1, wherein face model includes astructure of a face, the structure including a geometric mesh orcollection of geometric objects.
 3. The device of claim 1, wherein theone or more processors further execute the instructions to provide areal-time stream of at least one face associated with the real-time facemodel, the real-time stream being a video chat.
 4. The device of claim1, wherein the one or more processors further execute the instructionsto map the real-time face model to control levers on an animationrigging.
 5. The device of claim 1, wherein the manipulation includesmodifying visual parameters associated with a face.
 6. The device ofclaim 1, wherein the real-time face model includes at least two offacial contours, minute data points, or main data points.
 7. The deviceof claim 1, wherein the plurality of face parameters are animationparameters associated with a particular movement and feature of at leastone face.
 8. The device of claim 7, wherein the animation parametersinclude at least two of a movement of eyes, a movement of a nose, amovement of a jaw, a movement of a face bone structure, a movement ofeyebrows, or a movement of lips.
 9. The device of claim 1, wherein thereal-time face model includes audio data.
 10. The device of claim 9,wherein the device is operable such that the audio data is used tocreate an inference of how the manipulation is performed.
 11. The deviceof claim 1, wherein the correlation map matches up audio associated withone or more of phonetics, intonations, or emotions, with the one or moreface nodal points.
 12. The device of claim 1, wherein the device isoperable such that receiving the real-time face model requires less datausage than receiving real time video frames.
 13. The device of claim 1,wherein the device is operable such that the real-time face model isfurther refined using historical data or additional data associated withat least one face.
 14. The device of claim 1, wherein the device isoperable such that the real-time face model is automatically receivedbased on a network latency threshold or a dropped packet thresholdcondition in place of additional video frames.
 15. The device of claim1, wherein the device is further operable to receive a second real-timeface model associated with a third party individual or character,receive second real-time face nodal points, and manipulate the secondreal-time face model based on the second real-time face nodal points.16. A method, comprising: receiving, using a processor, a real-time facemodel, wherein the real-time face model includes one or more face nodalpoints and a plurality of face parameters; receiving, using theprocessor, real-time face nodal points, including additional one or moreface nodal points; and manipulating, using the processor, the real-timeface model based on the real-time face nodal points; wherein theplurality of face parameters is correlated and wherein the correlationoccurs synchronously; wherein the real-time face model includes an imagedepth map, an audio map, and a correlation map.
 17. A computer programproduct comprising computer executable instructions stored on anon-transitory computer readable medium that when executed by aprocessor instruct the processor to: receive a real-time face model,wherein the real-time face model includes one or more face nodal pointsand a plurality of face parameters; receive real-time face nodal points,including additional one or more face nodal points; and manipulate thereal-time face model based on the real-time face nodal points; whereinthe plurality of face parameters is correlated and wherein thecorrelation occurs synchronously; wherein the real-time face modelincludes an image depth map, an audio map, and a correlation map.
 18. Adevice, comprising: a non-transitory memory storing instructions; andone or more processors in communication with the non-transitory memory,wherein the one or more processors execute the instructions to: receivea real-time face model, wherein the real-time face model includes one ormore face nodal points; receive real-time face nodal points, includingadditional one or more face nodal points; and manipulate the real-timeface model based on the real-time face nodal points; wherein the deviceis operable such that the real-time face model is automatically receivedbased on a network latency threshold or a dropped packet thresholdcondition in place of additional video frames.
 19. A device, comprising:a non-transitory memory storing instructions; and one or more processorsin communication with the non-transitory memory, wherein the one or moreprocessors execute the instructions to: receive a real-time face model,wherein the real-time face model includes one or more face nodal pointsand audio data; receive real-time face nodal points, includingadditional one or more face nodal points; and manipulate the real-timeface model based on the real-time face nodal points; wherein the deviceis operable such that the audio data is used to create an inference ofhow the manipulation is performed.